The inferred user preference descriptions are used to define prompts for generating responses in the future.

While conceptually simple, we show that in practice, this reward modeling requires careful algorithmic considerations around model architecture and reward scaling.

We further explored an alternative loss function that is closer to CLIP's original objective.

The ability to collect a large dataset of human preferences from text-to-image users is usually limited to companies, making such datasets inaccessible to the public. To address this issue, we create a web app that enables text-to-image users to generate images and specify their preferences. Using this web app we build Pick-a-Pic, a large, open dataset of text-to-image prompts and real users'

Sequential recommendation aims to recommend the next item that matches a user's

Recommender systems have become ubiquitous across a wide range of fields, such as ecommerce, media consumption (including movies, books, music, news, etc.), social networks, finance, and many others, due to their effectiveness in identifying relevant items or content among numerous choices [1, 2]. Traditionally, recommender systems, largely based on collaborative filtering techniques, have focused on recommending individual (or "atomic") items, such as movies or books, by understanding users' preferences for these individual items. However, in certain application domains, recommending "composite" items (i.e., combinations of atomic items) represents a very important capability. For illustration, consider a clothing/fashion recommender system, where we want to recommend "outfits" - combinations of tops (t-shirts, shirts, sweaters) and bottoms (pants, skirts, shorts) - to users. In such a case, multiple fashion items in a recommended outfit ideally have to match both functionally and stylistically, which may require domain expertise (e.g., on things like style compatibility) beyond individual preferences. Another key challenge for such recommender systems is that a given user's personal preference for a composite item may not directly translate to the user's personal preferences for the underlying atomic items and vice versa.

We thank the reviewers for thoughtful feedback. "researchers working on pairwise comparisons and preference learning should find this paper to be interesting and Furthermore, we note that we also plan to make our code available as soon as the review period concludes. In our derivation, we pose the problem in a noiseless environment only for simplicity. For similar reasons, we also did not compare our method against algorithms utilizing different models of preference. As with any recommender system, practical considerations are important.